System and method for relating a passive sensor to a geographic environment

ABSTRACT

A method and system are provided for relating a passive sensor (14) to a geographic environment (42, 44). The passive sensor (14) senses an image (40) of the geographic environment (42, 44). At least one feature (46) is extracted (26) from the image (40). At least one feature is generated (30) from map data (28) representative of the geographic environment. At least one extracted feature is related (22, 32) to at least one generated feature, such that the passive sensor (14) is related to the geographic environment (42, 44).

This is a continuation of application Ser. No. 07/982,810, filed Nov.30, 1992.

TECHNICAL FIELD OF THE INVENTION

This patent application relates in general to image processing and inparticular to a method and system for relating a passive sensor to ageographic environment.

BACKGROUND OF THE INVENTION

Where a sensor senses an image of a geographic environment, automatictarget recognition ("ATR") is assisted by relating the sensor to thegeographic environment. The sensor can be either a passive sensor suchas a FLIR sensor or an active sensor such as a LADAR or RADAR sensor. Apassive sensor emits relatively little energy and primarily detectsenergy, such as black body radiation, emitted by a source other than thesystem to which the passive sensor belongs. A passive sensor is usefulfor measuring a velocity perpendicular to the sensor's line of sight,such as a velocity of a target object along a horizon between a sky anda planet's surface.

By comparison, an active sensor emits significant energy and primarilydetects a reflection of its own emitted energy from the geographicenvironment. Active sensors are useful for typical "ranging" systemsthat determine a three-dimensional model of a geographic environment'stopography. For example, an active sensor is useful for measuring arange of a target object away from the sensor, according to the traveltime of a reflected energy pulse. Also, an active sensor is useful formeasuring a velocity parallel to the sensor's line of sight, such as avelocity of a target object toward or away from the sensor, according toa frequency change of the reflected energy pulse.

Although an active sensor is practical for a diverse range ofenvironmental conditions, a passive sensor advantageously consumes lesspower and is more covert. Previous techniques typically fail to relate aposition and attitude of a passive sensor to a geographic environment.Accordingly, previous techniques typically fail to sufficiently reduceuncertainties concerning a target object's appearance in an image sensedby a passive sensor.

Thus, a need has arisen for a method and system for relating a passivesensor to a geographic environment, in which a position and attitude ofthe passive sensor are related to the geographic environment. Also, aneed has arisen for a method and system for relating a passive sensor toa geographic environment, in which uncertainties are reduced concerninga target object's appearance in an image sensed by the passive sensor.

SUMMARY OF THE INVENTION

In a method and system for relating a passive sensor to a geographicenvironment, the passive sensor senses an image of the geographicenvironment. At least one feature is extracted from the image. At leastone feature is generated from map data representative of the geographicenvironment. At least one extracted feature is related to at least onegenerated feature, such that the passive sensor is related to thegeographic environment.

It is a technical advantage of the present invention that a position andattitude of the passive sensor are related to the geographicenvironment.

It is another technical advantage of the present invention thatuncertainties are reduced concerning a target object's appearance in animage sensed by the passive sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, and theadvantages thereof, reference is now made to the following descriptionstaken in conjunction with the accompanying drawings, in which:

FIG. 1 is a block diagram of a navigation system, including processcircuitry for relating a passive sensor to a geographic environmentaccording to the preferred embodiment;

FIG. 2 is an exemplary image sensed by the passive sensor of ageographic environment;

FIG. 3 illustrates a discontinuity contour boundary superimposed overthe image of FIG. 2; and

FIG. 4 illustrates multiple intraterrain discontinuity contourboundaries superimposed over the image of FIG. 2.

DETAILED DESCRIPTION OF THE INVENTION

The preferred embodiment of the present invention and its advantages arebest understood by referring to FIGS. 1-4 of the drawings, like numeralsbeing used for like and corresponding parts of the various drawings.

FIG. 1 is a block diagram of a navigation system, indicated generally at10. System 10 includes process circuitry, indicated by dashed enclosure12, for relating a passive sensor 14 to a geographic environmentaccording to the preferred embodiment. In an exemplary embodiment,navigation system 10, process circuitry 12, and passive sensor 14 arecarried aboard an air vehicle (not shown).

An inertial navigation system ("INS") 16 of system 10 is coupled to aflight controller 18 of system 10, to a platform 20 of system 10, and tocontrol logic 22 of process circuitry 12. INS 16 outputs navigationalinformation to flight controller 18, to platform 20, and to controllogic 22. The navigational information includes an initial estimate of aposition and attitude of the air vehicle relative to the geographicenvironment. In response to the navigational information, and inresponse to a stored mission flight plan 24, flight controller 18controls movement of the air vehicle. Moreover, in response to thenavigational information, platform 20 substantially stabilizes passivesensor 14 relative to the geographic environment.

Passive sensor 14 senses an image of the geographic environment. Passivesensor 14 is coupled to a feature extractor 26 of process circuitry 12,and passive sensor 14 outputs the sensed image to feature extractor 26.In response to the sensed image, feature extractor 26 extracts at leastone feature from the sensed image. Such a feature is referred to as anextracted feature. Alternatively, such a feature can be referred to asan observed feature or as a scene feature. In exemplary embodiments, theextracted feature is a range discontinuity extracted by featureextractor 26 in response to intensity or color differences within thesensed image.

Storage circuitry 28 of system 10 stores digital data representative ofa three-dimensional map of the geographic environment. Storage circuitry28 is coupled to a feature generator 30 of processor circuitry 30, andstorage circuitry 28 outputs the digital map data to feature generator30. In response to the digital map data, feature generator 30 generatesat least one feature from the map data. Such a feature is referred to asa generated feature. Alternatively, such a feature can be referred to asa synthetic feature. In exemplary embodiments, the generated feature isa range discontinuity generated by feature generator 30 in response tothe map data.

Control logic 22 is coupled to feature extractor 26, to featuregenerator 30, and to a feature comparator 32 of process circuitry 12.Accordingly, in response to the sensed image from passive sensor 14, tothe digital map data from storage circuitry 28, and to the navigationalinformation from INS 16, process circuitry 12 relates at least oneextracted feature to at least one generated feature, so that passivesensor 14 is related to the geographic environment. From control logic22, INS 16 inputs an estimated position and attitude of passive sensor14 relative to the geographic environment.

In response to the estimated position and attitude from control logic22, INS 16 updates its navigational information. In this manner, system10 advantageously reduces drift error of INS 16. Moreover, in principle,performance specifications of INS 16 can be greatly relaxed where nointerludes occur during which system 10 might be inoperative, possiblyas when flying across water.

Process circuitry 12 estimates the position and attitude of passivesensor 14 relative to the geographic environment by analyzingsignificant range discontinuities, such as by delineating adiscontinuity contour boundary between objects at significantlydifferent ranges away from passive sensor 14. For example, where ashorter range target object is static relative to a longer range targetobject, and where both target objects move relative to a sensor, theshorter range target object exhibits greater movement than the longerrange target object in a sequence of images sensed by the sensor. Thisprinciple is demonstrated by optical flow of both target objects in thesequence of images.

Accordingly, process circuitry 12 delineates a discontinuity contourboundary to indicate a significant range discontinuity between objectsat significantly different ranges away from passive sensor 14. In anexemplary embodiment where the geographic environment includes a sky anda planet's surface, process circuitry 12 delineates a horizon betweenthe sky and the planet's surface to indicate a significant rangediscontinuity between the sky and the planet's surface, which are atsignificantly different ranges away from passive sensor 14. Normally, ahorizon discontinuity contour boundary has sharp contrast and awell-defined location.

FIG. 2 is an exemplary image, indicated generally at 40, sensed bypassive sensor 14 of a geographic environment including a sky 42 and aplanet's surface 44. Image 40 is sensed by passive sensor 14 from anactual pose defined in terms of position and attitude. Referring to FIG.3, in the exemplary embodiment, feature extractor 26 extractssignificant range discontinuities to delineate an observed horizon 46between sky 42 and surface 44, in response to intensity and colordifferences within image 40.

Feature generator 30 generates significant range discontinuities todelineate a synthetic horizon between the sky and the planet's surface,in response to the map data from storage circuitry 28 as viewed by ahypothetical observer from an assumed pose. Feature comparator 32 inputsthe synthetic horizon from feature generator 30 and inputs the observedhorizon from feature extractor 26. Feature comparator 30 determines adifference between the synthetic horizon and the observed horizon. Inresponse to the difference, feature comparator 30 outputs an errorfunction value ρ to control logic 22. In an exemplary embodiment, theerror function value ρ is determined according to a linear least squarescost function.

In response to the error function value ρ, control logic 22 adjusts theassumed pose used by feature generator 30. Feature generator 30generates a new synthetic horizon as viewed by a hypothetical observerfrom the assumed pose as adjusted by control logic 22. Featurecomparator 32 outputs a new error function value ρ indicating adifference between the new synthetic horizon and the observed horizon.In response to the new error function value ρ, control logic 22readjusts the assumed pose used by feature generator 30.

In this manner, process circuitry 12 correlates the observed horizonwith multiple synthetic horizons. Each of the synthetic horizonscorresponds to a respective assumed pose and to a respective errorfunction value ρ. Control logic 22 selects the assumed poses to becentered at an initial resolution around a position and attitudeindicated by navigational information from INS 16. The initialresolution is a function of an estimated reliability of navigationalinformation from INS 16. More particularly, the initial resolution isselected so that the domain of assumed poses has a 99% probability ofencompassing the actual pose of passive sensor 14. The position andattitude indicated by navigational information from INS 16 should accordwith the nominal position and attitude indicated by mission flight plan24. Accordingly, position and attitude are known within rather narrowlimits throughout the mission, so that computational demands of processcircuitry 12 are practical.

In response to the synthetic horizons' respective error function valuesρ, control logic 22 selects one of the synthetic horizons matching mostclosely with the observed horizon. The respective assumed pose of thisselected synthetic horizon is referred to as the first selected pose.Then, process circuitry 12 again correlates the observed horizon with anew set of multiple synthetic horizons, corresponding to respectiveassumed poses centered at twice the initial resolution around the firstselected pose. In response to the new set of synthetic horizons'respective error function values ρ, control logic 22 selects one of thenew set of synthetic horizons matching most closely with the observedhorizon. The respective assumed pose of this selected synthetic horizonis referred to as the second selected pose. From control logic 22, INS16 inputs the second selected pose to advantageously update itsnavigational information.

In generating significant range discontinuities to delineate a synthetichorizon, feature generator 30 renders a two-dimensional image as viewedby a hypothetical observer from an assumed pose, in response to thethree-dimensional map data stored in storage circuitry 28. Rendering iscomputationally demanding and is performed iteratively. Accordingly,feature generator 30 preferably includes high speed graphics processinghardware, such as a SILICON GRAPHICS IRIS 4D/70GT.

Such processing hardware further includes built-in features for readilydetermining significant range discontinuities from the map data. Forexample, feature generator 30 includes a z-buffer for storing rangesfrom the hypothetical observer to each pixel of the renderedtwo-dimensional image. Using the z-buffer, feature generator 30 readilydetermines a rate of change of range for each pixel over the entirerendered two-dimensional image. For a particular pixel, featuregenerator 30 determines a rate of change of range by differencing thez-buffer range value for the particular pixel with the average ofz-buffer range values for pixels adjacent to the particular pixel.

Feature generator 30 extracts significant range discontinuities byidentifying a histogram segmentation threshold for the rates of changeof range. By applying the identified histogram segmentation threshold,feature generator 30 identifies a set of pixels having significant ratesof change of range. The synthetic horizon is delineated by theidentified set of pixels.

Image dimensions of a target object can be estimated from informationconcerning slant range and sensor optics. Moreover, important terrainfeatures such as tree lines, roads and bridges are practicallyidentified for a more diverse range of environmental conditions whenapproximate positions and attitudes relative to map data are provided inadvance. Accordingly, relating a passive sensor to a geographicenvironment is applicable to autonomous guidance of robotic airvehicles, particularly for calibrating a Global Positioning System("GPS").

Accurate determination of sensor position and attitude can affect ATR inseveral ways. By establishing approximate slant range as a function ofpixel location, the image dimensions of a specific target object can beestimated. Uncertainty of viewing aspect is diminished by knowing thevertical look angle. These constraints are beneficial because theyreduce the number of degrees-of-freedom to be resolved from the imagealone.

Another benefit is that a "smart search" can be conducted, so that asearch for target objects is focused in regions where target objects aremost likely to be found. These regions are determined by tactical andphysical considerations and are closely associated with terraintopography and such features as tree lines, roads, and bridges.

Due to accuracy considerations, a two tier search strategy is expedient.In the first tier, the map data guides a search for terrain features ofinterest. These are detected by appropriate pattern recognitiontechniques. There is relatively little ambiguity as to feature locationbecause detection occurs directly in the image. In the second tier ofthe strategy, ATR algorithms are applied within regions of the imagedelineated by feature boundaries. Feature detection provides valuablecontextual information. For example, a moving object on a road is likelyto be a wheeled vehicle, aligned with the road, and pointed in thedirection of travel. Moreover, autonomous navigation readilyaccommodates contemporary intelligence, such as, "Tanks spotted headingnorth from map coordinates . . . ". System 10 supports ATR by providingnavigational accuracy.

An important design decision is the choice of features which link thereal world of the image to the virtual world of the map data. A goodfeature preferably satisfies a number of requirements:

Reliably extractable from image

Computable from the map data

Sensitive to changes of any component of sensor pose

Robust, so that it is practical over a wide range of environmentalconditions

Commensurate with the resolution of the map data

Sparsely distributed in scene

Amenable to efficient hardware implementation.

Since features from the image and map data originate from differentsources, it is preferable to choose attributes that are directlycomparable. Normally, the image contains a myriad of detail notrepresented in the map data. Much of this detail is subject tounpredictable temporal vicissitudes. The problem tends to diminish withthe physical size of the attribute.

Options for feature selection depend on the complexity of the map data.Several levels of map complexity are possible: (a) digital terrainelevation ("DTED") maps having a list of elevations as a function ofpoints on Earth's surface, (b) digital feature elevation ("DFAD") mapshaving qualitative feature descriptions such as road locations added toDTED maps, and (c) elevation maps combined with texture map overlays.Such texture map overlays are preferably taken from recent aerial photoreconnaissance. DTED maps are preferable based on simplicity andtemporal stability over time. A disadvantage of DTED maps is that theirapplication is limited to locales with suitable topography.

Topography is readily sensed from a special kind of image called a"depth map." In a depth map, range-to-object information is stored ateach pixel location. However, it is difficult to get accurate depth mapsfrom passive imagery. Optical flow fails to achieve sufficientresolution under some conditions of interest. Moreover, it is frequentlyimpractical to quickly determine optical flow. Accordingly, rangediscontinuity is the preferred feature, because sensor requirements areless stringent by defining the feature in terms of qualitative, ratherthan quantitative, range information. Range discontinuity is defined asa locus of contiguous points delineating a boundary between objects atsignificantly different ranges.

System 10 detects features in response to intensity factors. Boundariesare normally visible in an image on the basis of contrast, color ortexture. The process is greatly aided by a priori knowledge of aboundary's approximate location, size and shape from feature generator30. Advantageously, navigational errors are constrained, since INS 16updates its navigational information throughout the mission in responseto estimated positions and attitudes from process circuitry 12.

In a large number of scenes, the dominant range discontinuity is thehorizon. Consistently strong earth/sky contrast appears both during theday and at night under clear weather conditions in standard FLIRimagery. Horizon contours frequently exhibit qualities of being extendedand of varying considerably in depth relative to an observer. It isnormally practical to navigate in response to such a horizon contour.

Preferably, platform 20 adjusts passive sensor 14 to substantiallyoffset vehicle roll. Uncompensated roll is preferably accommodated bysystem 10. Notably, the attitude component of sensor pose affects theimage through rotations and translations and can be modeled as a lineartransformation. These are dealt with internally by feature comparator32. All contours sent to feature comparator 32 are resampled forconstant arc length between samples.

Preferably, features promote strong perspective effects. Such featurestend to have large angular extent in the image and to include elementsover a wide span of range. Resolution is preferably high to permitdiscerning of subtle changes. Control logic 22 analyzes the array ofassumed poses to determine the assumed pose exhibiting least nonlineardistortion.

FIG. 4 illustrates multiple intraterrain discontinuity contourboundaries 48a-g superimposed over image 40 of FIG. 2. Accordingly, itshould be understood that system 10 is also able to relate passivesensor 14 to the geographic environment in response to multipleintraterrain contours other than the horizon. In doing so, processcircuitry 12 selects the most promising features from the map data.Process circuitry 12 performs the following steps:

Step 1. Render image from map data for assumed pose;

Step 2. Transform z-buffer data to invert nonlinear mapping and restoretrue z-coordinates;

Step 3. (optional). Process transformed data pixel-by-pixel to convertfrom z-coordinates to slant ranges;

Step 4. Process to locate and quantify range discontinuities;

Step 5. Construct histogram of range discontinuity magnitudes and selectthreshold so that a predetermined fraction of range discontinuitymagnitudes exceed the threshold;

Step 6. Apply threshold and reject weak range discontinuities;

Step 7. Segment non-rejected contours according to a connected objectdetection algorithm;

Step 8. Rank segmented contours by length and retain N longest, where Nis a specified number.

Process circuitry 12 performs Step 2 because the z-buffer values are notthe z-coordinates themselves, but instead are a function of thez-coordinates. The output of Step 2 depends on the choice of clippingplanes in rendering and is inverted to achieve the suitable data. AtStep 3, process circuitry 12 converts z-coordinates into slant ranges.While desirable in principle, Step 3 is computationally demanding withlittle advantage; accordingly, Step 3 is optional. At Step 4, processcircuitry 12 determines the distribution of range discontinuitymagnitudes in the image.

From the distribution, at Step 5 process circuitry 12 selects athreshold so that a predetermined fraction (e.g., 10%) of rangediscontinuity magnitudes exceed the threshold. Referring to the outputof Step 5 shown in FIG. 4, in an exemplary embodiment, feature extractor26 extracts significant range discontinuities to delineate multipleintraterrain discontinuity contour boundaries 48a-g, in response tointensity and color differences within image 40. Notably, observedhorizon 46, being the strongest discontinuity, is also included.

At Step 6, process circuitry 12 applies the threshold from Step 5 toreduce the number of range discontinuity elements under consideration.At Step 7, process circuitry 12 uses a connected object algorithm toassign a single unique label to each contour. At Step 8, processcircuitry 12 ranks each labelled contour according to its length inimage 40. All but a specified number N of longest contours are rejected,so that the longest N contours form the set of selected features. Aftera feature contour is defined, it is optionally extendable to includeslightly weaker adjoining elements.

Intraterrain discontinuities are extractable from the image according toany of several possible techniques. By knowing the shape and location ofa feature contour a priori, an exemplary technique applies snakes asactive contour models. According to such an exemplary technique, acomponent of the snake's energy function is minimum where the snakeconforms to the expected shape. The difficulties presented by localminima are substantially addressed by the small uncertainty concerningthe feature contour's shape and location.

Although the present invention and its advantages have been described indetail, it should be understood that various changes, substitutions andalterations can be made therein without departing from the spirit andscope of the invention as defined by the appended claims.

What is claimed is:
 1. A system for relating a passive sensor to ageographic environment, the passive sensor being operable to sense animage of the geographic environment, comprising:storage circuitry forstoring map data having predetermined levels of complexity to relatewith the image and representative of the geographic environment; andprocess circuitry coupled to said storage circuitry and to the passivesensor and operable to:extract at least one image range discontinuityfrom the image; generate at least one map range discontinuity from saidmap data; and relate said at least one said extracted image rangediscontinuity to said at least one said generated map rangediscontinuity with said predetermined levels of complexity, such thatthe passive sensor is related to the geographic environment.
 2. Thesystem of claim 1, wherein said process circuitry is operable to relatea position and attitude of the passive sensor to the geographicenvironment.
 3. The system of claim 2, wherein said system furthercomprises an inertial navigation system coupled to said processcircuitry for determining navigational information, wherein saidinertial navigation system is operable to update said navigationalinformation in response to said related position and attitude.
 4. Thesystem of claim 1, wherein said range discontinuity comprises adiscontinuity contour boundary within the image between objects of thegeographic environment at significantly different ranges away from thepassive sensor.
 5. The system of claim 1, wherein said rangediscontinuity comprises an intraterrain discontinuity contour boundarywithin the image.
 6. The system of claim 1, wherein said extracted imagerange discontinuity comprises an observed horizon extracted from theimage.
 7. A method relating a passive sensor to a geographicenvironment, comprising the steps of:sensing an image of the geographicenvironment with the passive sensor; extracting at least one image rangediscontinuity from the image; generating at least one map rangediscontinuity from map data having predetermined levels of complexity torelate to the image and representative of the geographic environment;and relating at least one said extracted image range discontinuity to atleast one said generated map range discontinuity with said predeterminedlevels of complexity, such that the passive sensor is related to thegeographic environment.
 8. The method of claim 7, wherein said relatingstep comprises the step of relating a position and attitude of thepassive sensor to the geographic environment.
 9. The method of claim 8,wherein said method further comprises the step of updating navigationalinformation of an inertial navigation system in response to said relatedposition and attitude.
 10. The method of claim 9, wherein saidextracting step comprises the step of delineating a discontinuitycontour boundary within the image between objects of the geographicenvironment at significantly different ranges away from the passivesensor.
 11. The method of claim 10, wherein said delineating stepcomprises the step of delineating an intraterrrain discontinuity contourboundary within the image.
 12. The method of claim 7, wherein saidextracting step comprises the step of extracting an observed horizonfrom the image.